fflate
High performance (de)compression in an 8kB package
Why fflate?
fflate
(short for fast flate) is the fastest, smallest, and most versatile pure JavaScript compression and decompression library in existence, handily beating pako
, tiny-inflate
, and UZIP.js
in performance benchmarks while being multiple times more lightweight. Its compression ratios are often better than even the original Zlib C library. It includes support for DEFLATE, GZIP, and Zlib data. Data compressed by fflate
can be decompressed by other tools, and vice versa.
In addition to the base decompression and compression APIs, fflate
supports high-speed ZIP file archiving for an extra 3 kB. In fact, the compressor, in synchronous mode, compresses both more quickly and with a higher compression ratio than most compression software (even Info-ZIP, a C program), and in asynchronous mode it can utilize multiple threads to achieve over 3x the performance of virtually any other utility.
| pako | tiny-inflate | UZIP.js | fflate |
---|
Decompression performance | 1x | Up to 40% slower | Up to 40% faster | Up to 40% faster |
Compression performance | 1x | N/A | Up to 25% faster | Up to 50% faster |
Base bundle size (minified) | 45.6kB | 3kB (inflate only) | 14.2kB | 8kB (3kB for inflate only) |
Decompression support | ✅ | ✅ | ✅ | ✅ |
Compression support | ✅ | ❌ | ✅ | ✅ |
ZIP support | ❌ | ❌ | ✅ | ✅ |
Streaming support | ✅ | ❌ | ❌ | ✅ |
GZIP support | ✅ | ❌ | ❌ | ✅ |
Supports files up to 4GB | ✅ | ❌ | ❌ | ✅ |
Doesn't hang on error | ✅ | ❌ | ❌ | ✅ |
Dictionary support | ✅ | ❌ | ❌ | ✅ |
Multi-thread/Asynchronous | ❌ | ❌ | ❌ | ✅ |
Streaming ZIP support | ❌ | ❌ | ❌ | ✅ |
Uses ES Modules | ❌ | ❌ | ❌ | ✅ |
Demo
If you'd like to try fflate
for yourself without installing it, you can take a look at the browser demo. Since fflate
is a pure JavaScript library, it works in both the browser and Node.js (see Browser support for more info).
Usage
Install fflate
:
npm i fflate
Import:
import * as fflate from 'fflate';
import { gzipSync } from 'fflate';
If your environment doesn't support ES Modules (e.g. Node.js):
const fflate = require('fflate');
If you want to load from a CDN in the browser:
<script src="https://unpkg.com/fflate@0.8.2"></script>
<script src="https://cdn.jsdelivr.net/npm/fflate@0.8.2/umd/index.js"></script>
<script type="module">
import * as fflate from 'https://cdn.skypack.dev/fflate@0.8.2?min';
</script>
If you are using Deno:
import * as fflate from 'https://cdn.skypack.dev/fflate@0.8.2?min';
If your environment doesn't support bundling:
import * as fflate from 'fflate/esm/browser.js';
import * as fflate from 'fflate/esm';
And use:
const massiveFileBuf = await fetch('/aMassiveFile').then(
res => res.arrayBuffer()
);
const massiveFile = new Uint8Array(massiveFileBuf);
const notSoMassive = fflate.zlibSync(massiveFile, { level: 9 });
const massiveAgain = fflate.unzlibSync(notSoMassive);
const gzipped = fflate.gzipSync(massiveFile, {
filename: 'aMassiveFile.txt',
mtime: '9/1/16 2:00 PM'
});
fflate
can autodetect a compressed file's format as well:
const compressed = new Uint8Array(
await fetch('/GZIPorZLIBorDEFLATE').then(res => res.arrayBuffer())
);
const decompressed = fflate.decompressSync(compressed);
Using strings is easy with fflate
's string conversion API:
const buf = fflate.strToU8('Hello world!');
const compressed = fflate.compressSync(buf, { level: 6, mem: 8 });
const decompressed = fflate.decompressSync(compressed);
const origText = fflate.strFromU8(decompressed);
console.log(origText);
If you need to use an (albeit inefficient) binary string, you can set the second argument to true
.
const buf = fflate.strToU8('Hello world!');
const compressedString = fflate.strFromU8(
fflate.compressSync(buf),
true
);
const decompressed = fflate.decompressSync(
fflate.strToU8(compressedString, true)
);
const origText = fflate.strFromU8(decompressed);
console.log(origText);
You can use streams as well to incrementally add data to be compressed or decompressed:
let outStr = '';
const gzipStream = new fflate.Gzip({ level: 9 }, (chunk, isLast) => {
outStr += fflate.strFromU8(chunk, true);
});
gzipStream.ondata = (chunk, final) => { ... }
gzipStream.push(chunk1);
gzipStream.push(chunk2);
...
gzipStream.push(lastChunk, true);
console.log(outStr);
const deflateStream = new fflate.Deflate((chunk, final) => {
console.log(chunk, final);
});
const utfEncode = new fflate.EncodeUTF8((data, final) => {
deflateStream.push(data, final);
});
utfEncode.push('Hello'.repeat(1000));
utfEncode.push(' '.repeat(100));
utfEncode.push('world!'.repeat(10), true);
const inflateStream = new fflate.Inflate();
inflateStream.ondata = (decompressedChunk, final) => { ... };
let stringData = '';
const utfDecode = new fflate.DecodeUTF8((data, final) => {
stringData += data;
});
const dcmpStrm = new fflate.Decompress((chunk, final) => {
console.log(chunk, 'was encoded with GZIP, Zlib, or DEFLATE');
utfDecode.push(chunk, final);
});
dcmpStrm.push(zlibJSONData1);
dcmpStrm.push(zlibJSONData2, true);
console.log(JSON.parse(stringData));
You can create multi-file ZIP archives easily as well. Note that by default, compression is enabled for all files, which is not useful when ZIPping many PNGs, JPEGs, PDFs, etc. because those formats are already compressed. You should either override the level on a per-file basis or globally to avoid wasting resources.
const zipped = fflate.zipSync({
'dir1': {
'nested': {
'你好.txt': fflate.strToU8('Hey there!')
},
'other/tmp.txt': new Uint8Array([97, 98, 99, 100])
},
'massiveImage.bmp': [aMassiveFile, {
level: 9,
mem: 12
}],
'superTinyFile.png': [aPNGFile, { level: 0 }],
'exec': [{
'hello.sh': [fflate.strToU8('echo hello world'), {
os: 3,
attrs: 0o755 << 16
}]
}, {
mtime: new Date('10/20/2020')
}]
}, {
level: 1,
mtime: new Date('1/1/1980')
});
const decompressed = fflate.unzipSync(zipped, {
filter(file) {
return file.name != 'massiveImage.bmp' && file.originalSize <= 10_000_000;
}
});
If you need extremely high performance or custom ZIP compression formats, you can use the highly-extensible ZIP streams. They take streams as both input and output. You can even use custom compression/decompression algorithms from other libraries, as long as they are defined in the ZIP spec (see section 4.4.5). If you'd like more info on using custom compressors, feel free to ask.
const zip = new fflate.Zip((err, dat, final) => {
if (!err) {
console.log(dat, final);
}
});
const helloTxt = new fflate.ZipDeflate('hello.txt', {
level: 9
});
zip.add(helloTxt);
helloTxt.push(chunk1);
helloTxt.push(chunk2, true);
const nonStreamingFile = new fflate.ZipPassThrough('test.png');
zip.add(nonStreamingFile);
nonStreamingFile.push(pngData, true);
zip.end();
const unzipper = new fflate.Unzip();
unzipper.register(fflate.UnzipInflate);
const neededFiles = ['file1.txt', 'example.json'];
unzipper.onfile = file => {
if (neededFiles.includes(file.name)) {
file.ondata = (err, dat, final) => {
console.log(dat, final);
};
console.log('Reading:', file.name);
console.log('Compressed size', file.size);
console.log('Decompressed size', file.originalSize);
file.start();
}
};
unzipper.push(zipChunk1);
unzipper.push(zipChunk2);
unzipper.push(zipChunk3, true);
As you may have guessed, there is an asynchronous version of every method as well. Unlike most libraries, this will cause the compression or decompression run in a separate thread entirely and automatically by using Web (or Node) Workers. This means that the processing will not block the main thread at all.
Note that there is a significant initial overhead to using workers of about 50ms for each asynchronous function. For instance, if you call unzip
ten times, the overhead only applies for the first call, but if you call unzip
and zlib
, they will each cause the 50ms delay. For small (under about 50kB) payloads, the asynchronous APIs will be much slower. However, if you're compressing larger files/multiple files at once, or if the synchronous API causes the main thread to hang for too long, the callback APIs are an order of magnitude better.
import {
gzip, zlib, AsyncGzip, zip, unzip, strFromU8,
Zip, AsyncZipDeflate, Unzip, AsyncUnzipInflate
} from 'fflate';
const terminate = gzip(aMassiveFile, (err, data) => {
if (err) {
return;
}
console.log(data.length);
});
if (needToCancel) {
terminate();
}
zlib(aMassiveFile, { consume: true, level: 9 }, (err, data) => {
});
const gzs = new AsyncGzip({ level: 9, mem: 12, filename: 'hello.txt' });
let wasCallbackCalled = false;
gzs.ondata = (err, chunk, final) => {
if (err) {
console.error(err);
return;
}
wasCallbackCalled = true;
}
gzs.push(chunk);
console.log(wasCallbackCalled)
gzs.terminate();
zip({ f1: aMassiveFile, 'f2.txt': anotherMassiveFile }, {
level: 6
}, (err, data) => {
});
unzip(aMassiveZIPFile, (err, unzipped) => {
console.log(unzipped['data.xml']);
console.log(strFromU8(unzipped['data.xml']))
});
const zip = new Zip();
zip.ondata = (err, chunk, final) => { ... };
const exampleFile = new AsyncZipDeflate('example.json');
zip.add(exampleFile);
exampleFile.push(JSON.stringify({ large: 'object' }), true);
const exampleFile2 = new AsyncZipDeflate('example2.bmp', { level: 9 });
zip.add(exampleFile2);
exampleFile2.push(ec2a);
exampleFile2.push(ec2b);
exampleFile2.push(ec2c);
...
exampleFile2.push(ec2Final, true);
zip.end();
const unzip = new Unzip(stream => {
if (stream.name.endsWith('.json')) {
stream.ondata = (err, chunk, final) => { ... };
stream.start();
if (needToCancel) {
stream.terminate();
}
}
});
unzip.register(AsyncUnzipInflate);
unzip.push(data, true);
See the documentation for more detailed information about the API.
Bundle size estimates
The bundle size measurements for fflate
on sites like Bundlephobia include every feature of the library and should be seen as an upper bound. As long as you are using tree shaking or dead code elimination, this table should give you a general idea of fflate
's bundle size for the features you need.
The maximum bundle size that is possible with fflate
is about 31kB (11.5kB gzipped) if you use every single feature, but feature parity with pako
is only around 10kB (as opposed to 45kB from pako
). If your bundle size increases dramatically after adding fflate
, please create an issue.
Feature | Bundle size (minified) | Nearest competitor |
---|
Decompression | 3kB | tiny-inflate |
Compression | 5kB | UZIP.js , 2.84x larger |
Async decompression | 4kB (1kB + raw decompression) | N/A |
Async compression | 6kB (1kB + raw compression) | N/A |
ZIP decompression | 5kB (2kB + raw decompression) | UZIP.js , 2.84x larger |
ZIP compression | 7kB (2kB + raw compression) | UZIP.js , 2.03x larger |
GZIP/Zlib decompression | 4kB (1kB + raw decompression) | pako , 11.4x larger |
GZIP/Zlib compression | 5kB (1kB + raw compression) | pako , 9.12x larger |
Streaming decompression | 4kB (1kB + raw decompression) | pako , 11.4x larger |
Streaming compression | 5kB (1kB + raw compression) | pako , 9.12x larger |
What makes fflate
so fast?
Many JavaScript compression/decompression libraries exist. However, the most popular one, pako
, is merely a clone of Zlib rewritten nearly line-for-line in JavaScript. Although it is by no means poorly made, pako
doesn't recognize the many differences between JavaScript and C, and therefore is suboptimal for performance. Moreover, even when minified, the library is 45 kB; it may not seem like much, but for anyone concerned with optimizing bundle size (especially library authors), it's more weight than necessary.
Note that there exist some small libraries like tiny-inflate
for solely decompression, and with a minified size of 3 kB, it can be appealing; however, its performance is lackluster, typically 40% worse than pako
in my tests.
UZIP.js
is both faster (by up to 40%) and smaller (14 kB minified) than pako
, and it contains a variety of innovations that make it excellent for both performance and compression ratio. However, the developer made a variety of tiny mistakes and inefficient design choices that make it imperfect. Moreover, it does not support GZIP or Zlib data directly; one must remove the headers manually to use UZIP.js
.
So what makes fflate
different? It takes the brilliant innovations of UZIP.js
and optimizes them while adding direct support for GZIP and Zlib data. And unlike all of the above libraries, it uses ES Modules to allow for partial builds through tree shaking, meaning that it can rival even tiny-inflate
in size while maintaining excellent performance. The end result is a library that, in total, weighs 8kB minified for the core build (3kB for decompression only and 5kB for compression only), is about 15% faster than UZIP.js
or up to 60% faster than pako
, and achieves the same or better compression ratio than the rest.
Before you decide that fflate
is the end-all compression library, you should note that JavaScript simply cannot rival the performance of a native program. If you're only using Node.js, it's probably better to use the native Zlib bindings, which tend to offer the best performance. Though note that even against Zlib, fflate
is only around 30% slower in decompression and 10% slower in compression, and can still achieve better compression ratios!
What about CompressionStream
?
Like fflate
, the Compression Streams API provides DEFLATE, GZIP, and Zlib compression and decompression support. It's a good option if you'd like to compress or decompress data without installing any third-party libraries, and it wraps native Zlib bindings to achieve better performance than what most JavaScript programs can achieve.
However, browsers do not offer any native non-streaming compression API, and CompressionStream
has surprisingly poor performance on data already loaded into memory; fflate
tends to be faster even for files that are dozens of megabytes large. Similarly, fflate
is much faster for files under a megabyte because it avoids marshalling overheads. Even when streaming hundreds of megabytes of data, the native API usually performs between 30% faster and 10% slower than fflate
. And Compression Streams have many other disadvantages - no ability to control compression level, poor support for older browsers, no ZIP support, etc.
If you'd still prefer to depend upon a native browser API but want to support older browsers, you can use an fflate
-based Compression Streams ponyfill.
Browser support
fflate
makes heavy use of typed arrays (Uint8Array
, Uint16Array
, etc.). Typed arrays can be polyfilled at the cost of performance, but the most recent browser that doesn't support them is from 2011, so I wouldn't bother.
The asynchronous APIs also use Worker
, which is not supported in a few browsers (however, the vast majority of browsers that support typed arrays support Worker
).
Other than that, fflate
is completely ES3, meaning you probably won't even need a bundler to use it.
Testing
You can validate the performance of fflate
with npm test
. It validates that the module is working as expected, ensures the outputs are no more than 5% larger than competitors at max compression, and outputs performance metrics to test/results
.
Note that the time it takes for the CLI to show the completion of each test is not representative of the time each package took, so please check the JSON output if you want accurate measurements.
License
This software is MIT Licensed, with special exemptions for projects
and organizations as noted below:
- SheetJS is exempt from MIT licensing and may
license any source code from this software under the BSD Zero Clause License